{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ML_in_Finance-1D-CNNs\n",
    "# Author: Matthew Dixon\n",
    "# Version: 1.0 (28.4.2020)\n",
    "# License: MIT\n",
    "# Email: matthew.dixon@iit.edu\n",
    "# Notes: tested on Mac OS X running Python 3.6.9 with the following packages:\n",
    "# tensorflow=2.0.0, keras=2.3.1, numpy=1.18.1\n",
    "# Citation: Please cite the following reference if this notebook is used for research purposes:\n",
    "# Bilokon P., Dixon M.F. and Halperin I., Machine Learning in Finance: From Theory to Practice, Springer Graduate Textbook Series, 2020. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Keras to implement a 1D convolutional neural network (CNN) for timeseries prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "from keras.layers import Conv1D, Dense, MaxPooling1D, Flatten\n",
    "from keras.models import Sequential"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.set_printoptions(threshold=25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the 1D CNN\n",
    "\n",
    "We create a simple convolutional neural network: a 1D convolutional layer, followed by a dense layer.\n",
    "\n",
    "It will allow us to predict the next value in a timeseries given an input sequence of length `window_size`\n",
    "\n",
    "The `filter_length` is the length (in time-steps) of the sliding window that gets convolved with each position along each instance. The difference between 1D and 2D convolution is that a 1D filter's \"height\" is fixed to the number of input timeseries (its \"width\" being `filter_length`), and it can only slide along the window dimension.  This is useful as generally the input timeseries have no spatial/ordinal relationship, so it's not meaningful to look for patterns that are invariant with respect to subsets of the timeseries.\n",
    "`nb_filter` is the number of such filters to learn (roughly, input patterns to recognize).\n",
    "\n",
    "The model can handle multivariate timeseries (with `nb_input_series` variables) and multiple (`nb_outputs`) prediction targets. Predicting future values of a timeseries means setting these equal to one another."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_CNN(window_size, filter_length,  nb_filter=4, nb_input_series=1, nb_outputs=1):\n",
    "    \"\"\"\n",
    "    window_size (int): number of observations in each input sequence\n",
    "    filter length (int): length of the convolutional layer's filters\n",
    "    nb_filter (int): number of filters learned in the convolutional layer\n",
    "    nb_input_series (int): number of features of the input timeseries (1 for a univariate timeseries)\n",
    "    nb_outputs (int): number of features being predicted (equal to nb_input_series \n",
    "        for predicting a timeseries at a set horizon)\n",
    "    \"\"\"\n",
    "    model = Sequential((\n",
    "        # The convolutional layer learns `nb_filter` filters (aka kernels), \n",
    "        # each of size `(filter_length, nb_input_series)`.  \n",
    "        # Its output will have shape `(None, window_size - filter_length + 1, nb_filter)` ,  \n",
    "        # i.e., for each position in the input timeseries, the activation of each filter at that position.\n",
    "        Conv1D(filters=nb_filter, kernel_size=filter_length, activation='relu', input_shape=(window_size, nb_input_series)),\n",
    "        Flatten(),\n",
    "        Dense(nb_outputs, activation='linear'), # For classification, a 'sigmoid' activation function would be used\n",
    "    ))\n",
    "    model.compile(loss='mse', optimizer='adam', metrics=['mae'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "CNN_model = make_CNN(window_size=50, filter_length=5, nb_filter=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input shape: (None, 50, 1)\n",
      "output shape: (None, 1)\n"
     ]
    }
   ],
   "source": [
    "print('input shape:', CNN_model.layers[0].input_shape)\n",
    "print('output shape:', CNN_model.layers[-1].output_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data preparation\n",
    "  \n",
    "We define a function to format a timeseries for training the neural network. It creates corresponding arrays of input sequences `X` and output values `y`. They have the same length as each other; the remaining dimensions must match the input and output layers of the model respectively:\n",
    "\n",
    "The `X` input to the model's `fit()` method should be a 3D array of shape `(n_instances, window_size, n_ts_variables)`; each instance being a 2D array of shape `(window_size, nb_input_series)`.  For example, for `window_size = 3` and `nb_input_series = 1` (a univariate timeseries), one instance could be `[[0], [1], [2]]`\n",
    "\n",
    "For each input instance, the output is a vector of size `nb_outputs`, usually the value(s) predicted to come after the last value in that input instance, i.e., the next value in the sequence. The `y` input to ``fit()`` should be an array of shape ``(n_instances, nb_outputs)``. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_timeseries_instances(timeseries, window_size):\n",
    "    # Convert 1D vectors to 2D column vectors\n",
    "    timeseries = np.atleast_2d(timeseries)\n",
    "    if timeseries.shape[0] == 1:\n",
    "        timeseries = timeseries.T \n",
    "    \n",
    "    if not 0 < window_size < timeseries.shape[0]:\n",
    "        raise ValueError('Please set 0 < window size < timeseries length')\n",
    "    \n",
    "    # `X `is the tensor containing the inputs for the model\n",
    "    # each row of `X` is a sequence of `window_size` observations from the timeseries\n",
    "    X = [timeseries[start:start + window_size] for start in range(0, timeseries.shape[0] - window_size)]\n",
    "    \n",
    "    # for training the model, the array's dimensions must match the input layer of the CNN\n",
    "    # that is, a 3D array of shape (timeseries.shape[0] - window_size, window_size, nof_ts_variables)\n",
    "    X = np.atleast_3d(np.array(X))\n",
    "    \n",
    "    # For each row of `X`, the corresponding row of `y` is the \n",
    "    # desired output -- in this case, the subsequent value in the timeseries \n",
    "    y = timeseries[window_size:]\n",
    "    \n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X:\n",
      "[[[ 1]\n",
      "  [ 1]\n",
      "  [ 2]\n",
      "  [ 3]\n",
      "  [ 5]]\n",
      "\n",
      " [[ 1]\n",
      "  [ 2]\n",
      "  [ 3]\n",
      "  [ 5]\n",
      "  [ 8]]\n",
      "\n",
      " [[ 2]\n",
      "  [ 3]\n",
      "  [ 5]\n",
      "  [ 8]\n",
      "  [13]]]\n",
      "y:\n",
      "[[ 8]\n",
      " [13]\n",
      " [21]]\n"
     ]
    }
   ],
   "source": [
    "X_fib, y_fib = make_timeseries_instances([1,1,2,3,5,8,13,21], 5)\n",
    "print('X:', X_fib, 'y:', y_fib, sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a toy timeseries and split it for training and testing the CNN:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "timeseries = np.arange(1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = make_timeseries_instances(timeseries, window_size=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input instance:\n",
      " [[42]\n",
      " [43]\n",
      " [44]\n",
      " ...\n",
      " [89]\n",
      " [90]\n",
      " [91]]\n",
      "output instance:\n",
      " [92]\n"
     ]
    }
   ],
   "source": [
    "i = 42\n",
    "print('input instance:\\n', X[i])\n",
    "print('output instance:\\n', y[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "test_ratio = 0.01 # In real life you'd usually want to use 0.2 - 0.5\n",
    "test_size = int(test_ratio * len(timeseries)) \n",
    "\n",
    "# the \"most recent\" values are used for testing the model to avoid look-ahead bias\n",
    "X_train, X_test, y_train, y_test = X[:-test_size], X[-test_size:], y[:-test_size], y[-test_size:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the dimensions of the arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(940, 50, 1), (10, 50, 1), (940, 1), (10, 1)]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[i.shape for i in [X_train, X_test, y_train, y_test]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can fit the model. Note that `validation_data` is not used to train the model, but allows you to monitor its out-of-sample performance during training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 940 samples, validate on 10 samples\n",
      "Epoch 1/25\n",
      "940/940 [==============================] - 8s 8ms/step - loss: 22348.4393 - mae: 55.2219 - val_loss: 346.1044 - val_mae: 18.6034\n",
      "Epoch 2/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 197.2894 - mae: 11.7657 - val_loss: 155.3413 - val_mae: 12.4630\n",
      "Epoch 3/25\n",
      "940/940 [==============================] - 4s 5ms/step - loss: 198.5208 - mae: 11.8470 - val_loss: 802.1042 - val_mae: 28.3210\n",
      "Epoch 4/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 180.0273 - mae: 11.2627 - val_loss: 280.3401 - val_mae: 16.7429\n",
      "Epoch 5/25\n",
      "940/940 [==============================] - 4s 5ms/step - loss: 176.2927 - mae: 11.1332 - val_loss: 557.3866 - val_mae: 23.6086\n",
      "Epoch 6/25\n",
      "940/940 [==============================] - 4s 5ms/step - loss: 154.2343 - mae: 10.4821 - val_loss: 297.4342 - val_mae: 17.2459\n",
      "Epoch 7/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 142.0517 - mae: 9.9618 - val_loss: 162.1052 - val_mae: 12.7316\n",
      "Epoch 8/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 129.6742 - mae: 9.6935 - val_loss: 38.3850 - val_mae: 6.1951\n",
      "Epoch 9/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 104.5651 - mae: 8.6912 - val_loss: 49.1556 - val_mae: 7.0107\n",
      "Epoch 10/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 80.3527 - mae: 7.5985 - val_loss: 440.4976 - val_mae: 20.9878\n",
      "Epoch 11/25\n",
      "940/940 [==============================] - 4s 5ms/step - loss: 81.2117 - mae: 7.5812 - val_loss: 1750.9572 - val_mae: 41.8441\n",
      "Epoch 12/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 65.6879 - mae: 6.6927 - val_loss: 1.3965 - val_mae: 1.1811\n",
      "Epoch 13/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 36.8580 - mae: 5.0898 - val_loss: 87.3978 - val_mae: 9.3485\n",
      "Epoch 14/25\n",
      "940/940 [==============================] - 5s 5ms/step - loss: 32.3399 - mae: 4.7796 - val_loss: 74.3055 - val_mae: 8.6199\n",
      "Epoch 15/25\n",
      "940/940 [==============================] - 5s 5ms/step - loss: 22.1845 - mae: 3.9360 - val_loss: 63.7460 - val_mae: 7.9840\n",
      "Epoch 16/25\n",
      "940/940 [==============================] - 4s 5ms/step - loss: 20.9390 - mae: 3.8322 - val_loss: 81.9014 - val_mae: 9.0498\n",
      "Epoch 17/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 32.3463 - mae: 3.8536 - val_loss: 12.3042 - val_mae: 3.5077\n",
      "Epoch 18/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 7.7764 - mae: 2.2857 - val_loss: 2.6304 - val_mae: 1.6218\n",
      "Epoch 19/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 3.3744 - mae: 1.4850 - val_loss: 5.9422 - val_mae: 2.4376\n",
      "Epoch 20/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 98.2356 - mae: 5.0022 - val_loss: 0.1773 - val_mae: 0.4211\n",
      "Epoch 21/25\n",
      "940/940 [==============================] - 4s 5ms/step - loss: 1.3045 - mae: 0.9617 - val_loss: 0.0440 - val_mae: 0.2096\n",
      "Epoch 22/25\n",
      "940/940 [==============================] - 4s 5ms/step - loss: 1.2369 - mae: 0.9082 - val_loss: 0.4608 - val_mae: 0.6788\n",
      "Epoch 23/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 1.3552 - mae: 0.9364 - val_loss: 14.4391 - val_mae: 3.7999\n",
      "Epoch 24/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 1.0375 - mae: 0.7601 - val_loss: 0.0644 - val_mae: 0.2538\n",
      "Epoch 25/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 0.7156 - mae: 0.5850 - val_loss: 3.9436 - val_mae: 1.9858\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x64a47a320>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CNN_model.fit(X_train, y_train, epochs=25, batch_size=2, validation_data=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can inspect the weights of the convolutional layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv1d_1 (Conv1D)            (None, 46, 4)             24        \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 184)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 185       \n",
      "=================================================================\n",
      "Total params: 209\n",
      "Trainable params: 209\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'conv1d_1/kernel:0' shape=(5, 1, 4) dtype=float32, numpy=\n",
       " array([[[-0.34088135, -0.07874674,  0.4062053 ,  0.41992363]],\n",
       " \n",
       "        [[-0.00244047, -0.03935403,  0.29828435, -0.25241745]],\n",
       " \n",
       "        [[-0.22607194, -0.03523868, -0.3957395 , -0.41727823]],\n",
       " \n",
       "        [[ 0.3158957 ,  0.2066541 ,  0.39682862, -0.3310415 ]],\n",
       " \n",
       "        [[-0.13993861, -0.4061464 , -0.19353136,  0.14599642]]],\n",
       "       dtype=float32)>,\n",
       " <tf.Variable 'conv1d_1/bias:0' shape=(4,) dtype=float32, numpy=array([-0.05061084,  0.        ,  2.90622   ,  0.        ], dtype=float32)>]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CNN_model.summary()\n",
    "\n",
    "CNN_model.get_layer('conv1d_1').weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making predictions with the model\n",
    "Get the predicted values for the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = CNN_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 990.          991.97485352]\n",
      " [ 991.          992.97735596]\n",
      " [ 992.          993.97973633]\n",
      " [ 993.          994.98217773]\n",
      " [ 994.          995.98461914]\n",
      " [ 995.          996.98706055]\n",
      " [ 996.          997.98950195]\n",
      " [ 997.          998.99188232]\n",
      " [ 998.          999.99438477]\n",
      " [ 999.         1000.99682617]]\n"
     ]
    }
   ],
   "source": [
    "print(np.column_stack((y_test, y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiple-input, multiple-output prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[   0    0    0    0]\n",
      " [   1    2    3    4]\n",
      " [   2    4    6    8]\n",
      " ...\n",
      " [ 997 1994 2991 3988]\n",
      " [ 998 1996 2994 3992]\n",
      " [ 999 1998 2997 3996]]\n"
     ]
    }
   ],
   "source": [
    "n_vars = 4\n",
    "mv_timeseries = np.array([i * np.arange(1000) for i in np.arange(1, n_vars + 1)]).T\n",
    "print(mv_timeseries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X[0]: \n",
      "[[  0   0   0   0]\n",
      " [  1   2   3   4]\n",
      " [  2   4   6   8]\n",
      " ...\n",
      " [ 47  94 141 188]\n",
      " [ 48  96 144 192]\n",
      " [ 49  98 147 196]]\n",
      "y[0]:\n",
      "[ 50 100 150 200]\n"
     ]
    }
   ],
   "source": [
    "mv_X, mv_y = make_timeseries_instances(mv_timeseries, 50)\n",
    "\n",
    "print('X[0]: ', mv_X[0], 'y[0]:', mv_y[0], sep='\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "test_ratio = 0.01 # In real life you'd usually want to use 0.2 - 0.5\n",
    "test_size = int(test_ratio * len(timeseries)) \n",
    "\n",
    "mv_X_train, mv_X_test = mv_X[:-test_size], mv_X[-test_size:] \n",
    "mv_y_train, mv_y_test = mv_y[:-test_size], mv_y[-test_size:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "mv_CNN_model = make_CNN(window_size=50, filter_length=5, nb_filter=4, nb_input_series=n_vars, nb_outputs=n_vars)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 940 samples, validate on 10 samples\n",
      "Epoch 1/25\n",
      "940/940 [==============================] - 9s 10ms/step - loss: 67570.3614 - mae: 70.6544 - val_loss: 429.2058 - val_mae: 18.6915\n",
      "Epoch 2/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 570.9748 - mae: 17.0400 - val_loss: 1157.8059 - val_mae: 29.1417\n",
      "Epoch 3/25\n",
      "940/940 [==============================] - 4s 5ms/step - loss: 651.1784 - mae: 18.1105 - val_loss: 379.6758 - val_mae: 17.3287\n",
      "Epoch 4/25\n",
      "940/940 [==============================] - 5s 5ms/step - loss: 567.5281 - mae: 17.1644 - val_loss: 2001.1688 - val_mae: 34.8229\n",
      "Epoch 5/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 596.8147 - mae: 17.6290 - val_loss: 964.9664 - val_mae: 27.7559\n",
      "Epoch 6/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 542.8998 - mae: 16.8306 - val_loss: 166.5247 - val_mae: 10.1677\n",
      "Epoch 7/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 413.6827 - mae: 14.8041 - val_loss: 2493.7627 - val_mae: 42.8001\n",
      "Epoch 8/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 321.5546 - mae: 13.1654 - val_loss: 365.8237 - val_mae: 16.3420\n",
      "Epoch 9/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 419.4454 - mae: 15.2131 - val_loss: 35.9475 - val_mae: 5.6044\n",
      "Epoch 10/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 232.1903 - mae: 11.2450 - val_loss: 307.6538 - val_mae: 14.3550\n",
      "Epoch 11/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 218.1219 - mae: 10.7669 - val_loss: 155.4201 - val_mae: 9.7333\n",
      "Epoch 12/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 1200.8580 - mae: 18.3666 - val_loss: 2700.4934 - val_mae: 49.0041\n",
      "Epoch 13/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 1680.7057 - mae: 18.3256 - val_loss: 263.2259 - val_mae: 12.8719\n",
      "Epoch 14/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 68.7152 - mae: 6.1090 - val_loss: 7.3022 - val_mae: 2.1528\n",
      "Epoch 15/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 58.3898 - mae: 5.6050 - val_loss: 54.1291 - val_mae: 5.9204\n",
      "Epoch 16/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 45.7635 - mae: 4.9655 - val_loss: 74.8121 - val_mae: 8.1314\n",
      "Epoch 17/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 29.5542 - mae: 3.9824 - val_loss: 1671.6571 - val_mae: 40.2824\n",
      "Epoch 18/25\n",
      "940/940 [==============================] - 5s 5ms/step - loss: 340.0878 - mae: 7.6176 - val_loss: 96.0178 - val_mae: 8.2130\n",
      "Epoch 19/25\n",
      "940/940 [==============================] - 5s 5ms/step - loss: 316.9279 - mae: 9.5790 - val_loss: 21.1099 - val_mae: 4.2915\n",
      "Epoch 20/25\n",
      "940/940 [==============================] - 4s 4ms/step - loss: 16.2405 - mae: 2.8794 - val_loss: 5.7342 - val_mae: 1.9679\n",
      "Epoch 21/25\n",
      "940/940 [==============================] - 4s 5ms/step - loss: 328.2318 - mae: 9.5278 - val_loss: 1.4304 - val_mae: 0.9187\n",
      "Epoch 22/25\n",
      "940/940 [==============================] - 4s 5ms/step - loss: 31.7229 - mae: 3.8444 - val_loss: 87.3611 - val_mae: 8.9864\n",
      "Epoch 23/25\n",
      "940/940 [==============================] - 5s 5ms/step - loss: 1188.6710 - mae: 14.9445 - val_loss: 1.7880 - val_mae: 1.1058\n",
      "Epoch 24/25\n",
      "940/940 [==============================] - 5s 5ms/step - loss: 11.0744 - mae: 2.5114 - val_loss: 22.2642 - val_mae: 4.7168\n",
      "Epoch 25/25\n",
      "940/940 [==============================] - 4s 5ms/step - loss: 179.7705 - mae: 7.4043 - val_loss: 112.1095 - val_mae: 10.5271\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x64b0e7198>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mv_CNN_model.fit(mv_X_train, mv_y_train, epochs=25, batch_size=2, validation_data=(mv_X_test, mv_y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "mv_y_pred = mv_CNN_model.predict(mv_X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "true: [ 990 1980 2970 3960] \tpred: [ 980.3531 1968.6608 2958.1584 3950.858 ]\n",
      "true: [ 991 1982 2973 3964] \tpred: [ 981.3445 1970.6503 2961.1492 3954.8552]\n",
      "true: [ 992 1984 2976 3968] \tpred: [ 982.3357 1972.6409 2964.1401 3958.8518]\n",
      "true: [ 993 1986 2979 3972] \tpred: [ 983.32697 1974.6309  2967.1316  3962.8486 ]\n",
      "true: [ 994 1988 2982 3976] \tpred: [ 984.31824 1976.621   2970.1228  3966.8457 ]\n",
      "true: [ 995 1990 2985 3980] \tpred: [ 985.3092 1978.6108 2973.1133 3970.8428]\n",
      "true: [ 996 1992 2988 3984] \tpred: [ 986.3005 1980.6011 2976.1047 3974.8394]\n",
      "true: [ 997 1994 2991 3988] \tpred: [ 987.2918 1982.591  2979.0962 3978.836 ]\n",
      "true: [ 998 1996 2994 3992] \tpred: [ 988.28284 1984.5813  2982.0872  3982.833  ]\n",
      "true: [ 999 1998 2997 3996] \tpred: [ 989.274  1986.5712 2985.0784 3986.8303]\n"
     ]
    }
   ],
   "source": [
    "for i in range(test_size):\n",
    "    print('true:', mv_y_test[i], '\\tpred:', mv_y_pred[i])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
